DSGEM: Dual scene graph enhancement module‐based visual question answering

Visual Question Answering (VQA) aims to appropriately answer a text question by understanding the image content. Attention-based VQA models mine implicit relationships between objects according feature similarity, which neglects explicit objects, for example, relative position. Most Scene Graph-based exploit positions or visual construct scene graph, while they suffer from semantic insufficiency of edge relations. Besides, graph modality is often ignored in these works. In this article, novel Dual Graph Enhancement Module (DSGEM) proposed that exploits relevant external knowledge simultaneously two interpretable structures and modalities, makes reasoning process more logical precise. Specifically, authors respectively build textual graphs with help commonsense syntactic structure, explicitly endows specific semantics each relation. Then, enhancement modules are propagate involved structural guide interaction (nodes). Finally, embed such existing introduce relation ability. Experimental results on both V2 OK-VQA datasets show DSGEM effective compatible various architectures.